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Description 

General Line of Sight Stabilization 

System 

Federal Research Statement 

[0001] This invention was made with U.S. Government support 

under NASA NAS 13-03014. The U.S. Government has a 

royalty-free right to the use of this invention. 
Background of Invention 

[0002] Field of the Invention 

[0003] jhe invention relates to optical systems and their ele- 
ments, and to image orientation and stabilization. More 
particularly, the invention is directed to line of sight image 
stabilization by way of movable reflective structures such 
as mirrors. In particular, the invention concerns hyper- 
spectral and other optical passive and active imaging sys- 
tems that can benefit from three axes image stabilization 
as well as orientation, such as airborne and spaceborne 
(off- nadir viewing) imaging systems. The apparatus of the 



invention shows particular utility for use with airborne and 
spaceborne pushbroom hyperspectral and, with forward 
motion compensation, ultraspectral imaging systems, 
agricultural and military applications, or in any other ap- 
plication requiring three-axes line of sight image stabi- 
lization and orientation. 
[0004] Background 

[0005] The background of this invention involves imaging sys- 
tems as well as image orientation and stabilization sys- 
tems. Multi spectral, hyperspectral, ultraspectral, and other 
imaging systems are well known in the art, as are image 
orientation and stabilization systems. Typical image stabi- 
lization systems reduce the magnitude of changes in the 
attitude, orientation, or location of light arriving at a de- 
tector array in a passive imaging system, or light depart- 
ing the source on an active airborne system, in one or 
more axes. Such changes are affected by redirecting in- 
coming or departing light or moving the detector array or 
source. 

[0006] Certain imaging systems have particular requirement for 
image orientation and stabilization. For example, hyper- 
spectral imaging systems are often of a pushbroom type 
(described below), benefiting from three-axis stabiliza- 



tion. Ultraspectral imaging systems benefit from three- 
axis stabilization plus forward motion control (hereinafter 
"FMC") in order to allow extended un-perturbed exposure 
times. There is considerable prior art to address these 
stabilization requirements. 
[0007] Common in the art are hyperspectral imaging systems 
(both of the pushbroom and whiskbroom type) designed 
to acquire an image of a narrow strip in the cross-track 
flight direction (typically one pixel along-track and a fixed 
width of N-pixels cross-track). Often, such imaging sys- 
tems rely on the forward motion of the imaging system 
carrier (for example, an airplane, helicopter, satellite, un- 
manned aerial vehicle (UAV), or other such equipment) to 
"sweep" forward. A subsequent narrow strip image is ac- 
quired after a fixed amount of time for forward motion. 
This process can repeat for any length of time. The multi- 
ple narrow strip images are "stitched" together to form a 
complete image, of the fixed N-pixel width, and of a 
length, equal to the number of snapshots taken while in 
forward motion. . Such an imaging system is typically re- 
ferred to as a pushbroom imaging system. Pushbroom 
and other imaging systems are well known in the art. Such 
system can benefit from image orientation and stabiliza- 



tion as each subsequent image preferentially must align 
closely with its predecessor image. When the carrier plat- 
form (hereinafter "CP") executes attitude changes due to 
wind gusts or air turbulence, the orientation of the narrow 
image strip changes and the scene may be under or over 
sampled. The orientation and stabilization system should 
maintain each strip in the cross track direction and cen- 
tered on the nadir view of the platform. Without image 
stabilization, the series of images are nearly impossible to 
stitch to form a single, contiguous image of the scene. 

[0008] Approximately twenty years ago, in one of the earliest 
pushbroom system patents, Minott (United States Patent 
Number 4,407,563) acknowledged this issue. Minott ex- 
plained that the pitch, roll, and yaw of a satellite had to be 
known in order to compensate for these movements in the 
post-image processing. While Minott dealt with aberra- 
tions due to the use of beam splitters, the problem of 
compensating for movement in three dimensions has re- 
mained critical for multispectral imaging. 

[0009] | n the intervening years, many techniques have been de- 
veloped for improving image stabilization. These tech- 
niques fall into several general categories. The first are 
the entirely gimbal-mounted systems. A gimbal is simply 



a device with two equally perpendicular and intersecting 
axes of rotation, giving free angular movement in two di- 
rections. When an imaging apparatus is mounted in a 
gimbal, the imaging system can target any object in a very 
large field of regard (hereinafter "FOR"), often encompass- 
ing half a sphere or more. Moreover, such a system can 
remain pointed at a single target and can then compen- 
sate for attitude changes in three axes: pitch, roll, and 
yaw. Gimbals are often seen in police and search and res- 
cue helicopter mounted flood lamps and television-style 
cameras that can point at, and track, any object on the 
ground. 

[0010] a disadvantage of the gimbal systems, however, is that 
while they can compensate for pitch, roll, and yaw move- 
ments, unimportant to tracking systems, television cam- 
eras, and flood lamps, the gimbal cannot maintain orien- 
tation. Orientation changes occur either when the carrier 
(typically for tracking setups, a helicopter) approaches or 
recedes from a target in an off-nadir trajectory, or when 
the carrier has a speed relative to the tracked object. For 
example, the orientation of an image captured by a cam- 
era mounted in a gimbal underneath a helicopter changes 
as the helicopter pursues a ground vehicle and catches up 



to the ground vehicle (in order to fly above the target so 
that ground units may visually locate the target by ob- 
serving the more visible helicopter). For a typical gimbal- 
mounted television camera, this orientation change is 
unimportant: the television viewer simply sees the target 
from various angles. The important feature is that the tar- 
get is somewhat centered in the frame, but the direction 
that the target is moving relative to the "bottom of the 
screen" is irrelevant. 
1 ] In a pushbroom imaging system or in an automated target 
identification system (comparing the target image to a 
known database of targets), however, these orientation 
changes result in columns of the scanned ground grid not 
being aligned or the identification failing for lack of a 
steady target image. In the pushbroom case, instead of 
obtaining nice columns of image, intensive post-imaging 
processing is needed to compensate for the orientation 
changes, and often, at best, only the center of the image 
will be useful, though likely no part of the image will be 
useful. Where the CP performs a nose pitch upward, cer- 
tain strips of image will be missing completely and the 
lost data will not be recoverable. In the target identifica- 
tion case, if the target cannot remain steady in the image, 



comparison software will not have a sufficiently detailed 
image from which to make database comparisons (for ex- 
ample, identifying a vehicle, person, or other target of in- 
terest). In Ultraspectral Imaging systems, the orientation 
problem makes registering a series of interferograms im- 
practicable. 

[0012] a further disadvantage of gimbal systems is that the in- 
strumentation used for imaging must fit within a gimbal. 
That means that special equipment is needed for use with 
a gimbal, or at the least, the gimbal may need to be ex- 
tremely large to accommodate a large imaging apparatus. 
Larger equipment also requires larger motors, which in 
turn increases the total bulkiness, cost, and weight of 
such systems. 

[0013] a second technique to provide image stabilization is the 
use of a scanning mirror, located between the imaging 
system optics (the lens) and the detector array. Such a 
scanning mirror can be very small and compact. By pivot- 
ing the scanning mirror, the image can be redirected to 
the detector array quickly and effectively. This provides an 
obvious advantage to cost and system size. However, 
there are several disadvantages to such systems. The first 
is that a single scanning mirror can only compensate for 



two axes of movement. The scanning mirror, like the gim- 
bal system (and in fact, the mirror can be mounted in a 
very small gimbal within the camera optics), can compen- 
sate for pitch and roll, but not yaw. Yaw, or a change in 
orientation of the CP, cannot be changed with a single 
mirror. A further disadvantage of this art is that such a 
scanning mirror must be built into the optics, which then 
requires special arrangement of the fore optics of the 
sensor. A scanning mirror of this type cannot be added to 
an existing optical system without significant retrofit cost 
and technology. This is a costly consideration for existing 
imaging systems. 
[0014] Finally, the third, and heretofore most accurate of the typ- 
ical image stabilization techniques is the use of a large 
"flatbed" on which to mount the optics system. The 
flatbed is motorized to move in all directions, allowing the 
imaging system to be tilted to compensate for pitch, roll, 
and yaw. In fact, by way of a series of complex calcula- 
tions, appropriate pitch, roll, and yaw compensations can 
combine to further provide FMC, as well. Although such 
flatbed image stabilizations systems appear to resolve the 
orientation and stabilization problem for pushbroom 
scanners, there are significant disadvantages to these 



systems. Such imaging system stabilization control must 
move the entire imaging system. In order to stabilize im- 
ages for large optical systems, the motors and entire 
flatbed system must be very robust. Such motors need to 
be able to quickly move a large mass, accurately, and in 
constantly changing directions. Such motors are very ex- 
pensive, thus making flatbed stabilization not very cost 
effective. If less expensive motors are used, the flatbed 
stabilization cannot respond to changes of direction as 
quickly and the image is less (or perhaps insufficiently) 
stabilized. 

[0015] M 0S t recently, several companies have began testing new 
stabilization systems, including combining a pivoting, 
gimbal-type system with a Dove Prism (NovaSol, in 
Hawaii, US) and combining two mirrors in the imaging 
system optics and further allowing the entire optics as- 
sembly to move in order to provide three axes of stabi- 
lization (Zeiss Optronik GmbH, Germany, US Patent Num- 
ber 6,370,329). The gimbal-based system continues to 
suffer from the limitations imposed by requiring the opti- 
cal system to fit within a gimbal, while the mirrored, 
moveable optics requires special imaging systems. These 
systems thus still do not provide for an economically fea- 



sible method of retrofitting image stabilization to existing 
systems, or provide compact, cost-effect image stabiliza- 
tion to new systems. 
[0016] Accordingly, there is a need in the art for a simple, effec- 
tive, and cost efficient image orientation and stabilization 
system that overcomes or avoids the above problems and 

limitations. 
Summary of Invention 

[0017] The invention comprises a General Line Of Sight Stabiliza- 
tion System (hereinafter referred to as "GLOSS system" or 
"GLOSSS" interchangeably) using two mirrors pivotally 
mounted to a fixed platform that can be used with exist- 
ing imaging systems while providing pitch, roll, and yaw 
compensation while maintaining image orientation. 

[0018] By deflecting only the photons of the scene (passive imag- 
ing) or source (active imaging), the inventive system 
avoids the need to stabilize the entire imaging sensor and 
optics system. The only mass to move is that of the two 
imaging system mirrors. This approach requires small 
motors with low torque, providing a less expensive, 
lightweight, and small image orientation and stabilization 
system. 

[0019] The invention further comprises hardware control means 



that allow predictive and responsive image stabilization to 
quickly and accurately compensate for typical platform in- 
stability without the use of large, expensive motors and 
components. The inventive system limits mechanized mo- 
tor control to the moving of two pivoted, and relatively 
small, mirrors. 

[0020] The invention monitors the CP attitude changes via an in- 
ertial measurement system, predicts the expected plat- 
form position for subsequent image acquisitions, and 
controls and positions the mirrors so as to provide the 
optimal orientation and stabilization for the subsequent 
image. The inertial measurements unit can be one a pre- 
existing one (common equipment on many airplanes, 
UAVs, satellites, or other such CPs) or can be specifically 
installed as a part of the inventive system. 

[0021] The invention additionally encompasses calculation in- 
structions and optionally calculation hardware for deter- 
mining the coupled movement of the two mirror system, 
making possible the object of using a compact, indepen- 
dent image orientation and stabilization system capable of 
working with a wide variety of imaging systems and carri- 
ers (helicopters, airplanes, satellites, and others). The in- 
vention also makes possible the object of upgrading and 



retrofitting existing image acquisition systems at minimal 
cost. 

[0022] a further object of the invention is to minimize the weight 
of the entire system, critical for use in space and weight 
constrained applications, including military and space- 
borne purposes. 

[0023] Additionally, an object of the invention is to provide a ori- 
entation and stabilization method for ultraspectral imag- 
ing, using the inventive mirror control method to provide 
forward motion compensation, allowing the imaging sys- 
tem to receive a stable, continuous, non-moving image 
for a sufficient time to acquire data for a relatively long 
exposure time as required for subsequent Fourier trans- 
formation. 

[0024] Further applying the inventive image stabilization system 
to other optical systems, it is an object of the present in- 
vention to provide image orientation and stabilization for 
active airborne laser altimeters and other active air- or 
spaceborne systems. Similarly, it is an object of the 
present invention to provide image orientation and stabi- 
lization to passive still cameras, television cameras, and 
any other optical systems that can benefit from three axes 
stable and consistently oriented images. 



[0025] The scope of application of the inventive apparatus is 

broad, as a number of alternative image stabilization and 
control schemes may suggest themselves to those skilled 
in the art as suitable for a wide variety of image stabiliza- 
tion applications, both for pushbroom hyperspectral sys- 
tems and for other general optical systems. These appli- 
cations include agricultural field scanning and spaceborne 

weather data acquisition, to name a few. 
Brief Description of Drawings 

[0026] The accompanying views of the drawings are incorporated 
in, and constitute a part of, this specification and illustrate 
one or more exemplary non-limiting embodiments of the 
invention, which, together with the description, serves to 
explain the principles of the invention. In the drawings: 

[0027] Figure 1 shows a schematic view of the current best mode 
of the inventive apparatus, including the two mirrors (one 
single-axis pivot mounted, the other gimbal mounted), 
the mounting platform, and a typical pushbroom hyper- 
spectral imaging system for purposes of clarity; and 

[0028] Figure 2 shows a flow chart of the operation of the mirror 

control system. 
Detailed Description 



[0029] The following detailed description illustrates the invention 
by way of example, not by way of limitation of the princi- 
ples of the invention. This description will clearly enable 
one skilled in the art to make and use the invention, and 
describes several embodiments, adaptations, variations, 
alternatives, and uses of the invention, including what are 
presently believed to be the best modes of carrying out 
the invention. 

[0030] | n this regard, the invention is illustrated in the several 

figures and is of sufficient complexity that the many parts, 
interrelationships, process steps, and sub-combinations 
thereof simply cannot be fully illustrated in a single 
patent-type drawing or table. For clarity and conciseness, 
several of the drawings show particular elements in 
schematic and omit other parts or steps that are not es- 
sential in that drawing to a description of a particular fea- 
ture, aspect, or principle of the invention being disclosed. 

[0031] The inventive GLOSS system is generally described by Fig- 
ure 1, which shows the current best mode application of 
the present invention. The system is comprised of two 
mirrors mounted to a platform (10) through small gimbals 
(20, 22). Mirror One (1) is mounted through a single axis 
pivot (20) (though a gimbal could be used, as well), while 



Mirror Two (2) is mounted through a two-axis gimbal 
(22). The platform can be solidly affixed to any optical 
imaging or projecting system, hyperspectral, ultraspectral, 
video or still camera, laser projection, or otherwise. Here, 
the exemplary optical system is a pushbroom hyperspec- 
tral system (30). The platform is then attached to the CP 
(not shown in the Figure). 
[0032] The first mirror is mounted to the platform by way of a 
one-axis pivot (20). The second mirror is mounted to the 
platform by way of a two-axis pivot (22). The mirrors are 
mounted such that in their nominal positions, the three 
axes are perpendicular to each other, and parallel to the 
CP's three axes of pitch, roll, and yaw. While the aircraft 
attitude is changing, the mirrors (1, 2) compensate for the 
attitude and orientation change, which will typically result 
in the mirror-axes no longer remaining parallel to the air- 
craft axes. The compensation for pitch, roll, or yaw may 
require mirror pivoting in a singular or coupled manner, 
depending on the specific attitude or orientation correc- 
tion. As the mirrors (1, 2) move, the reflections move and 
require further compensation, such that most changes in 
one mirror require coupled, and calculated, changes in the 
second mirror. Thus, a change in roll alone will require an 



adjustment to both mirrors (1, 2) to compensate. 

[0033] The mirrors (1, 2) are mounted in a plane with the imag- 
ing system (30), and that plane is parallel to the target 
ground, allowing the imaging system to "see" perpendicu- 
lar to the ground, looking down onto linear strips of 
ground. The linear strip of ground is extended perpendic- 
ular to the direction of travel of the plane. In this way, the 
pushbroom system (30) can make successive scans of the 
ground as the carrier moves forward. 

[0034] The mirrors (1, 2) are associated with an inertial measure- 
ment system (40) (hereinafter "IMS" or "IMU" for the "unit" 
itself) or an Attitude and Heading Reference System 
(hereinafter "AHRS"). Either of these devices contains three 
orthogonal gyrometers (42) (also commonly referred to as 
"gyros") that measure the CP rotation rates about the 
three axes. If the IMU (40) is not located at the roll center 
of the CP, then the calculation system must account for 
the IMU displacement from the origin. The IMU (40) is 
coupled to a computer (50) and the incoming data are 
used to determine the necessary motion for the two mir- 
rors (1, 2) to completely orient and stabilize the image. 
The computer (50) sends signals to the mirror control 
motors (again, not shown), which in turn, adjust the mir- 



rors (1, 2). As the mirrors (1, 2) are relatively low in mass 
and the compensations are small in size or angles, such 
compensations can be nearly instantaneous, providing 
very accurate image stabilization. 

[0035] | n the preferred embodiment, the mirrors move in a con- 
tinuous and smooth pattern predictively "mirroring" the 
aircraft motion. In this way, the mirrors are not subject to 
continuous "stop" and "go" commands, reducing the as- 
sociated accelerations and thus reducing the torque- 
requirement for the motors, allowing smaller and less ex- 
pensive motors. By setting the mirrors in a continuous 
motion of the same angular velocities and accelerations as 
the CP undergoes, the mirrors are in the right position for 
each snapshot of the sensor (noting that mirror motion is 
generally opposite to CP direction, with angular deflec- 
tions being about one half of the CP attitude changes). 

[0036] The objective of the GLOSSS is to have the mirror in the 
correct position at the time each frame is captured. The 
predictive mirror movements are derived from the IMS in- 
formation. From the axes rotation rates and integration 
over time, the actual angle of the CP is calculated. The 
calculations must compensate for system noise, drift, and 
temperature-induced errors in the gyrometers. Next, by 



knowing the aircraft angle over several time units and 
knowing the angular rotation rates, the location of the CP 
in the next time step can be predicted. Such a predictive 
algorithm can be obtained by a variety of means, from 
simple extrapolation to curve fitting using polynomial 
spline-fit or other such methods. More sophisticated pre- 
dictive tools can be employed, as well, including some 
that tend to compensate for system noise, such as a 
Kalman filter based tool. Once the proceeding location is 
predicted, the GLOSS system must compute the desired 
mirror positions to compensate for the CP predicted posi- 
tion, and cause the mirror motors to move at the proper 
rate to that position. 

[0037] Although the optimal motion for the mirrors is at a rate 
similar to that of the CP, in situations where the CP is 
suddenly caused to move along one of its axes (a gust of 
wind, a low pressure pocket, or otherwise), the proceeding 
location predictions may be incorrect, so the predictive al- 
gorithms must "catch up." In order to catch up, the GLOSS 
system must be capable of quickly deploying the mirrors 
with new instructions to compensate for the unexpected 
stabilization need. 

[0038] Figure 2 is a Flow Chart of the system control logic, as 



well as the mirror control logic to provide pitch, roll, and 
yaw corrections for the mode of use shown in Figure 1. As 
the table shows, the Mirror Control aspect of the Mirror 
Control System starts with the current mirror positions (at 
time t(n)). The mirror control system then activates the 
Motor Controller, which in turn activates the Mirror Motors 
and Servos, which feedback to the Motor Controller. This 
action is prompted by an instruction to establish a new 
mirror position for the proceeding time interval, time 
t(n+l). The new mirror position instruction is derived 
from the expected imaging sensor or CP orientation and 
position for time t(n + l), which is determined by the CP 
Position Predictor (hereinafter the "CPPP"). The CPPP takes 
input from the IMS and the previously predicted CP posi- 
tion and applies a Kalman (or other) filter to create a new 
prediction for time t(n+l). 
[0039] The Mirror Control system receives the sensor and CP lo- 
cation information, as well as the mirror orientation and 
location information, and the mirror axes status. This in- 
formation is added to the current mirror pointing and a 
ray tracing calculation is made. The line of sight for the 
imaging system is calculated using the same coordinate 
system as was used for the carrier and mirror positions. A 



coordinate transformation is applied (Euler Angles) and 
the resultant information is provided as the imaging sys- 
tem line of sight. This data is compared to the desired line 
of sight for the imaging system. If there is no difference, 
the system flags the mirrors as pointing appropriately. If 
there is a difference between the desired and actual line of 
sight, the mirror control system applies an optimization 
method to estimate a better mirror position. This new es- 
timate is fed to the mirror control system. 

[0040] By following the simple logic system described by Figure 
2, the mirrors can continuously provide properly oriented 
and stabilized images to the imaging system. 

[0041] industrial Applicability: 

[0042] | t j S c | ear t hat the image stabilization system of the 

present invention will have wide industrial applicability, 
not only to agricultural and military applications, but also 
to other applications where image stabilization is neces- 
sary or desired, including still photography, television, 
and other such applications. The inventive apparatus and 
method will further have great applicability in any circum- 
stance where image stabilization is desired to be added to 
existing imaging systems, or where space, weight, or cost 
are important factors. 



